ci: stop rebuilding what has not changed - #1150
Open
kvinwang wants to merge 4 commits into
Open
Conversation
`rust-checks` is the largest job in the repository at ~12 minutes, it is a required status check, and it was compiling the workspace from an empty target directory on every run. Every other Rust job in the repository is in the same state except two: the gateway suites, and `simulator-release.yml`, which has been using `Swatinem/rust-cache` all along. This uses the same action the same way. `workspaces: dstack` because the manifest is not at the repository root, and the action keys the cache on the lockfile it finds there.
…moved Two changes to the same job, both about work it did not need to do. The cache is the same one `rust-checks` just got; `sdk/rust` is a separate workspace from `dstack` and gets its own entry. The filter is a step, not a `paths:` on the workflow. `sdk-tests` is a required status check: a workflow skipped by `paths:` reports nothing at all, so the check never arrives and the pull request waits on it forever. Gating the expensive steps inside a job that always runs keeps the report and drops the work -- a documentation-only pull request costs a checkout and a diff. The pattern includes `dstack/` because `sdk/run-tests.sh` starts the simulator, which is built from that workspace: these suites exercise the agent's wire surface, not only the client libraries. The whole directory rather than the simulator's dependency closure, which is a dozen crates deep and would go stale the first time one of them moved.
Neither is a required status check, so a workflow-level `paths:` filter is safe here -- a pull request that skips them is not left waiting on a check that never arrives. Docker Build Check verifies that the three builder images still build and that their pinned package lists still match, which cannot change unless something under `dstack/` does. It is three jobs of roughly eight minutes each and it was running on every pull request, including documentation-only ones. The images also get a layer cache, scoped per image. It covers the pinned-package install and the toolchain setup and stops there: the cargo build cannot be cached, because the source arrives through `git clone` at DSTACK_REV inside the build rather than from the build context, so BuildKit has nothing to key it on. That is deliberate -- the revision is what the image records in /etc/.GIT_REV -- so the ceiling is the layers above the clone. The VMM UI build is a minute and only ever concerns `dstack/vmm/ui`.
Both matrix legs call `simulator_start`, which builds `dstack-guest-agent-simulator` out of the current tree. Same binary, same commit, compiled twice in parallel. The legs still build it themselves -- nothing here replaces that call or hands them a binary from elsewhere, so what they test is still what the commit produces. A `simulator` job runs first and populates the cargo cache under a key both legs restore, so the build they run finds its work already done. A skip switch in `simulator_build` would have been shorter and would have created a path where the binary under test did not come from the checkout, which is not worth a few minutes. The workflow also gains a `paths:` filter. `dstack/**` rather than the guest-agent directories alone: the closure reaches a dozen crates, and naming the obvious three would leave the rest silently uncovered.
kvinwang
force-pushed
the
ci/speed-up-pr-checks
branch
from
August 27, 2026 07:50
3f68750 to
d335ea9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A PR push currently costs about 120 runner-minutes. Most of it is the same
Rust workspace being compiled again, and jobs running on changes they cannot be
affected by.
Measured from the last 12 runs of each workflow, and from one push on #1147:
Analyze (rust)rust-checksrust.ymlsdk-testssdk.yamlkms/gateway/verifierdocker-build-check.ymlsdk-compat.yamlWhat this changes
Caching.
rust-checksis the largest job that gates every merge and it hadno cargo cache at all. Neither did
sdk-tests.simulator-release.ymlhas beenusing
Swatinem/rust-cacheall along, so this is the repository's own patternapplied to the jobs that need it most.
Filters.
rust.yml,sdk.yaml,sdk-compat.yaml,docker-build-check.ymland
vmm-ui.ymlhad nopaths:at all: a one-line documentation change ranroughly 50 minutes of Rust compilation.
rust-checksandsdk-testsare required status checks, so they cannottake a workflow-level
paths:— a skipped workflow reports nothing, the checknever arrives, and the pull request waits on it forever. Those two get the
filter as a step inside a job that always runs and always reports; the
expensive steps are gated on its output. The three that are not required take
an ordinary
paths:.One agent build instead of two. Both
sdk-compatlegs builtdstack-guest-agent-simulatorfrom the same commit, in parallel. Asimulatorjob now populates the cargo cache first and both legs restore it. They still
run the build themselves, so what they test is still what the commit produces.
Rust CodeQL — the largest job of all — is not here. Moving it needs a
repository setting changed in the same breath, so it is #1151.
What was checked
actionlintandprek run --all-filesare clean.The two gate patterns were tested against a file list rather than reasoned
about:
rust-checkssdk-testsdocs/deployment.mdREADME.mdos/mkosi/…dstack/gateway/src/main.rsdstack/ra-tls/src/lib.rssdk/python/…sdk/simulator/build.shrust-toolchain.tomlA push with no base SHA yields
relevant=true, so a non-PR build never skips.That table is also where a real bug surfaced. The
sdk-testspattern started assdk/only, which was wrong:sdk/run-tests.shcallssimulator_start, andthe simulator is built from the
dstackworkspace — these suites exercise theagent's wire surface, not just the client libraries, so a
dstack/change mustrun them. Both
sdk.yamlandsdk-compat.yamlnow name the whole directoryrather than the simulator's dependency closure: the closure is a dozen crates
deep and would go stale the first time one of them moved, and a filter that
quietly stops covering something is worse than one that occasionally runs when
it need not.
Not done here
fresh runner they compile Rust from cold every time. Fixing that means
cache_to: type=ghaon the compose builds; it belongs with those suitesrather than with the shared CI configuration.
rust-checkscompiles the workspace twice —cargo clippywith defaultfeatures, then
cargo test --all-features— because the two featureresolutions share few artifacts. Aligning them would also make clippy lint the
test code, which it currently does not (
--all-targetsis missing). Worthmeasuring first, since it may surface new lints.